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In everyday life, there are situations where the only way to communicate are 
emotions. EMOTICONS are the epitome of the same. This aspect of 
communication can also be used in emergency situations (terrorist attacks, 
hijacks) where the only way to communicate is by performing some 
extraordinary actions or through some emotions. Incorporating technology to 
the above mentioned circumstances the paper proposes a novel framework of 
detecting an emergency situation by retrieving information from emotions 
and eye blinks using sensor nodes. The proposed framework can be deployed 
in places (hotels, banks, airports etc.) which are more suspected to attacks. 
The framework takes input from real time parameters: eye blinks, emotions, 
heart rate. Based on behavioral changes, biological changes and physical 
changes the proposed framework extracts meticulous information. The 
proposed framework is further validated through implementation of a facial 
emotion recognition system that successfully recognizes various human 
emotions. The facial emotion recognition system of the proposed framework 
is compared with existing SVM technique in terms of accuracy, training and 
testing error. Accuracy with the proposed system is increased to 78.40% in 
comparison with existing SVM that is 75.37% and the training error is 
decreased to 0.004103 whereas with the existing SVM method training error 
is 0.008935. 
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1. INTRODUCTION 


In recent times many events of hostages and hijacks have happened and there is an urgent need of an 
automated system which can make the concerned agencies aware of the situation in time and thus help them 
to take necessary action. Wireless Sensor Networks can be used in attack prone areas to examine the situation 
and hence will help the concerned agencies to take appropriate action in time. 

The 26/11 Mumbai attacks were a planned series of attacks, when a group of 10-12 terrorists of 
Lashkar-e-Taiba, carried out a sequence of coordinated bombing and shooting that continued for four 
days [1], [2]. Densely populated areas like railway station and 5 star hotels (Hotel Taj) were their main 
targets. The attacks, which were heavily censured globally, began on 26" November and lasted until 29" 
November 2008, killing 164 people and wounding at least 300. A lot of damage had already been done even 
before the security agencies came to know what actually had happened. So if there would have been such 
automated systems, a lot would have been saved in terms of lives and public property. 
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Another such event of hostage includes hijacking of Indian Airline Flight 814 en route from 
Tribhuvan International Airport, Nepal to Indira Gandhi Airport, India on Friday, 24 December 1999. The 
most recent Uri attack was an inhuman attack by heavily armed terrorists in Uri area of Baramulla district, 
Jammu and Kashmir. It is considered as "the deadliest attack on security forces in last two decades”. No 
terrorist group has claimed responsibility for the same. 

In this paper we propose a novel framework for detecting an emergency situation which uses the 
intelligent information retrieved from pulse rate and eye blinks [3] using wireless sensor nodes as well as 
behavioral patterns [4], [5] exhibited by humans in response to any changes in the former parameters. To the 
best of our knowledge, such security framework that confirms emergency or intrusion detection on all given 
parameters does not exist. 

The parameters in response abnormal situation discussed here are: 

a. Eye blink pattern, 
b. Human Emotions, 
c. Heart rate 

For each parameter we intent to analyze the behavioral (emotion), biological (pulse rate) and 
physical (eye blink) pattern changes as measured by the systems deployed which can in future be used to 
install an emergency or intrusion warning system. 

Author in [6] have presented a novel approach to detect whether the eyes in an image (still) are 
closed or not and this approach has wide ranging applications in facial expression recognition, fatigue 
detection, and so on. A similar analysis in [7] has been discussed to detect eye blinks in a video sequence 
from a standard camera. Emotion is also one of the parameters which play an important part in our proposed 
framework. In [8] a regional hidden Markov model (RHMM) is proposed for recognizing facial expressions 
in video sequences. Facial action units are described for eyebrows, eyes and mouth regions by RHMMs. 
Proposed technique has outperformed other existing methods when tested with extended Cohn-Kanade 
database. Author in [9] presents an Automatic Face Analysis (AFA) system to inspect facial expressions 
taking into consideration both permanent facial features (brows, eyes, mouth) and momentary facial features 
(deepening of facial furrows). 


2. RESEARCH METHOD 

Sensor Nodes as known for the monitoring abilities such as accuracy and sensitivity are used as the 
baseline for the proposed framework. Figure 1 describes the proposed Framework. 

The intelligent information collected from standalone parameters, will act as input to the system 
which after performing calculations will decide whether it has crossed the threshold or not. Upon crossing the 
threshold an alarm or buzzer or some message will be transmitted to the concerned authorities. The sensors 
nodes will be deployed to extract the profound information from the given parameters, thus can be used as 
forerunner for provoking the alarm messages. 

a. Eye blink 

This attribute is of utmost importance as far as our proposed framework is concerned. Average eye 
blinking rate is around 12-19 per minute. (Intentional Eye blinks > 25 >Abnormal). 

Proposed framework uses the sensor placed in the line of sight of the eye which continuously studies 
the blink pattern. Sensors used will draw a curve to indicate a change in eye blink [10], [11] rate. 

b. Human emotions 

Emotion is a cognitive state that is generated impulsively rather than through responsive effort and 
is accompanied by physiological changes which are familiar to the outer world. Some emotions which 
humans face in everyday life are worry, surprise, confuse, happy etc. Human vision can practice emotion as 
coupled with personality, temperament, and mood. Computer Vision tries to follow the human vision by 
analyzing digital image as input. Sensors deployed will continuously study the human emotions. A database 
of emotions with an instance of time will be maintained with sensor nodes. 

c. Heart rate 

Heart rate is the number of times a heart beats in a minute. Heart rate [12] varies from person to 
person depending on one’s age, body size, fitness and heart condition. The same also depends on whether the 
person is running, standing, sitting, moving or is under influence of some medication. Emotions can also 
increase the heart rate. A healthy heart beats 60 to 100 times in a minute and can spike up due to stress, 
overexertion or nervousness. Our proposed framework will continuously monitor the heart rate. 

The proposed framework unfolds the entire structure of the process combined together. Smart motes 
(nodes) will be installed as per the requirements. Three types of sensors (eye blink, emotion detection and 
heart rate sensors) will be deployed. The sensors will work independently and will transfer the sensed 
attributes to the smart server via smart gate where the actual feature selection will take place. Then after 
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some observation, calculation and comparisons, the final verdict (whether the threshold is reached or not) 
will be taken. If threshold is crossed, alarm will be raised. For example, if the eye blink sensor senses an 
unusual blink pattern or number, in addition to a Fear or Surprise emotion detected by the emotion detection 
sensor or an abnormal heart rate, the situation is an alarming one. Blink pattern and emotions can be 
calculated by still or moving images via specialized cctv’s. For heart rate concerned persons (bar tenders in 
hotels, bank managers, flight attendants) will be equipped with wireless heart rate sensors or straps. 


[N 
/ \ 
\ 

© 

© s Doe 

sensors 

© 

\ j lrrrrrs = 
VY Z 


mainserver 


AT 
° w 


alarm 


Figure 1. Proposed Framework for Exigency detection 


The proposed framework works in two phases. The first phase (analysis phase) takes input from eye 
blinks, emotions and heart rate. The second phase constitutes the classification phase. The reactions could be 
change in behavior or emotions or physiological over given time. In this paper a Perspicacious Information 
Retrieval technique which senses serious diversion of parameters from baseline values via sensors spanned 
across target zones is presented. The data is transmitted to a base station which is further processed at main 
server. The moment major deflection from normal is observed using the information gathered, sensors are 
trained to issue alarm signals as in when such abnormal activity is sensed. 


2.1. Implementing information retrieval technique 

Figure 2 shows how methodically Perspicacious Information Retrieval Technique (PIRT) is 
implemented at servers where messages stating unusual human emotion, pulse rate and blink pattern are 
received at base station. It is important to know that input values would be gathered from some trained staff 
(air hostess in airplanes, bartenders in hotels, some officials in banks etc) having the knowledge of how to 
react in an emergency situation. 

The proposed framework should be deployed in areas which are in direct line of sight of the 
concerned persons and will help generate alarm signals as soon as change in behavioral or blink patterns is 
sensed. Messages acquired are further refined in order to excerpt intelligent information from it. Figure 2 
presents a novel prognostic algorithm which takes as input sensed data from the preceding stages and figure 
out certain conditions on the data received. 

Various stages of PIRT are: 
a. Z1: Eye blink 

This standalone condition evaluates the Blink pattern (BP) and hence the numbers of Blinks in say 1 
minute. Average eye blinking rate is around 12-19 per minute. So an unusual number/pattern will be the 
indication that something abnormal is there. For a precise decision we are taking it 25 Blinks per minute. The 
value stated which approximately accounts to 25 is the lowest default value. As no signs of change in blink 
pattern are observed below value - No alarm will be raised. For getting the BRM (blink rate per minute), 
frames from the video sequence can be extracted and will be further processed for detecting face and hence 
the eye portion of the image. From the eye landmarks [7] detected from the image, the eye aspect ratio (EAR) 
that is used as an approximate of the eye opening/closing state can be calculated. The eye aspect ratio (EAR) 
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is computed between height and width of the eye. Characteristic ratio of the open eye has a small discrepancy 
among humans and it is fully uniform to a constant scaling of the image. As blinking is practiced by both 
eyes simultaneously, the EAR of the pair is averaged. The EAR is primarily constant for an open eye and is 
approximately zero for a closed eye. 
Pattern for condition Z1: 
{ Z1: calculate BRM 
if (BRM<25) 
no alert ; 
Else 
proceed to Z2 ( 2™ condition) } 


Change in 
pulse rate 
All three attributes are sensed 
Data analysis via prognostic 
algorithm 
Facet value analysis 


Exigency 
detected 


Change in 
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Change in 
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Prognostic Algorithm 


1. Obtain the blink pattern from the sensors (1 min 
duration). 


2. For a given pattern say number of blinks be BP 


2.1 If BP <= 25 
2.1.1 No alert generated-> Go to Step 1. 


2.2 While 25 < BP 
2.2.1 Change in Heart rate (HR) and 
Emotions (EM) are sensed 
2.2.1.1 if ((HR > NHR) or 
( EM= (Fear or Surprise))) 
Go to Step 3 
Else Go to Step 1 
//NHR= normal heart rate 


3. Generate Alert signals 


Figure 2. Perspicacious Information Retrieval Technique for exigency detection 


b. Z2: Heart rate 

This condition will be evaluated only if condition Z1 is true and will evaluate whether the Heart rate 
(HR) or pulse has crossed the normal defined rate. If it has, alarm will be generated; otherwise we proceed to 
check the next condition which is getting ORed with the current condition. 
Pattern for condition Z2: 

{ If (Z1) 

{ Z2: calculate HR //NHR Normal Heart Rate 

if (HR>NHR) 
generate Alert signals; 
else 
proceed to Z3; 

H 
c. Z3: Facial emotion 

This condition evaluates the emotion detected. If the resultant emotion detected is FEAR or 
SURPRISE, alarm signal will be generated (only if condition Z1 stands true) else we will proceed to first 
condition again. It is important to note that either Condition 2 (Z2), 3 (Z3) or both should be tested to 
generate warning messages (given Z1 is true). 
Pattern for condition Z3: 

{If (Z1) 

{Z3: Detect Emotion 
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if (Emotion== (Fear or Surprise)) 
generate Alert signals; 

else 
Safe state; 


H 


2.2. Facet value analysis 

The facet evaluation is done on the sensed data to further figure out conditions and generate alert 
signals whenever condition Z1, Z2 or Z3 evaluates to true/false in certain permutations or combination. 
Based on above analysis we validate a framework that can hatch alert signals based on messages received by 
the sensors deployed. Figure 3 shows some cause effect graphs which state that condition Z1 in conjunction 
with Z2 or Z3 or both is a necessary prerequisite for the framework to issue warning signal. 


Z1&Z28&Z3 Z18&Z28&-Z3 Z18&-Z2&-Z3 


~-Z1&Z2&Z3 ~Z1&Z2&~Z3 ~Z1&~Z28&Z3 ~Z1&-~Z2&-~Z3 


Figure 3. Cause graph based on condition Evaluated 


2.3 Emotion detection 

The paper presents a detailed analysis of one of the parameters of the proposed framework: Human 
Emotions [13], [14]. A Facial Emotion Recognition System is proposed that would recognize 7 emotions 
labeled Neutral, Angry, Disgust, Fear, Happy, Sad and Surprise using Matlab. The proposed system will 
extract emotions on the basis of some unique features. Figure 4 presents a novel algorithm for detecting 
emotions from an image. The image to be tested can be taken from a smart phone using an app IP Webcam 
which turns the android phone into wireless camera. 


3. RESULTS AND DISCUSSION 

The performance of proposed Facial emotion recognition system is evaluated and Table 1 
showcases the comparison between the proposed method and existing SVM during the training and testing 
phase. The proposed method is compared with existing SVM (support vector machine) technique in terms of 
accuracy, training and testing error. Accuracy and training error comparison between SVM and the proposed 
method is shown in Figure 5. Accuracy with the proposed method is increased to 78.40% in comparison with 
existing SVM that is 75.37%. Training error for the proposed method is decreased to 0.004103 whearas with 
the existing SVM method training error is 0.008935. 
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Algorithm for Emotion Detection L1 = cropL(10:20,10:30); //eye 
L2 = cropL(20:30,10:30); //nose 
e Preprocess the database (cohn-kanade) L3 = cropL(30:40, 10:30); //mouth 
Create train directory 
mkdir(‘Training_data\0'); IbpH1 = lbp(L1); //ocal binary pattern (eye) 
= IbpH2 = lbp(L2);//local binary pattern (nose) 
mkdir(‘Training_data\5'); IbpH3 = lbp(L3); //local binary pattern (mouth) 
mkdir('Training data\6'); IbpHist=[lbpH1,lbpH2,lbpH3];//LBP 
folder 0:= all neutral images 
folder 1:= all angry images //A histogram of the labeled image f(x, y) can be 
z defined as 
folder 6:= all surprise images H=} «yI {fix, y) =i} ,i=0,....n-1 
in which n is the number of different labels produced 
Test directory <= (25-30% images of train by the LBP operator and 
directory) I {A} = 1, A is true 
= 0, A is false. 
e Training and testing data matrix for concerned This histogram contains information about the 
database is prepared using local binary pattern. distribution of the local micro patterns, such as edges, 
for i=1:no_files spots and flat areas, over the whole image. 
filename = files {i}; 
Im = imread(filename); . (Training step) Train matrices are then used to 
Call next two steps for face detection, cropping (eye, train the classifier (neural network). Network is 
nose and mouth) and local binary histogram [15]. trained by applying training data and training 
trainX(i,:) = IbpHist; parameters (learning rate parameter (eta = 0.01) 
train (i,str2double(filename(19))+1) = 1; and number of epochs (nEpochs = 500), hidden 
Proceed to Training step. layers (H = 16)) as input which results in updated 
weights of the input and hidden layer with 
e Detect faces from the images using voila jones minimum error. 
algorithm [16] [17]. 
[X,Y,height,width] = violaJones(Image); To complete the emotion detection, practice the 
proposed system takes as input updated weights 
e The target face will be divided into three regions w and v (backward pass) from previous step and 
(eye, nose and mouth) testing points X (LBP features of image to be 
cropL = imcrop(Image,[X, Y,height,width]); tested) and concludes with the family of the data 
cropL = imresize(cropL,[40,40]); as anticipated by the network. 


Figure 4. Algorithm for detecting emotions 


ACCURACY TRAINING ERROR 


SVM PROPOSED METHOD 


Figure 5. Accuracy and training error comparison between SVM and the Proposed Method 


Figure 6 shows the image which system accepts as input and Figure 7 depicts the resultant emotion 
(neutral) being detected. Two different resultant emotions (surprise, happy) are also shown in Figure 8 and 
Figure 9. Training error is shown in Figure 10 which reduces with each iteration. 


Int J Elec & Comp Eng, Vol. 8, No. 4, August 2018 : 2433 — 2441 


Int J Elec & Comp Eng ISSN: 2088-8708 O 2439 


Table 1. Comparison between Proposed Method and SVM 


SVM (Training) Proposed Method SVM Proposed Method 
(Training) (Testing) (Testing) 
Total Number of Instances 370 370 264 264 
Correctly Classified Instances 370 369 199 207 
Incorrectly Classified Instances 0 1 65 57 
Mean absolute error 0.2041 0.0025 0.2098 0.0722 
Kappa statistic 1 0.9962 0.6321 0.6791 
Root mean squared error 0.3006 0.0229 0.3099 0.2281 
Relative absolute error 100.4048 % 1.2529 % 106.4637 % 36.6319 % 
Root relative squared error 94.4743 % 7.1946 % 100.5206 % 73.994 % 


Figure 8. Emotion 2 Figure 9. Emotion 3 
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Figure 10. Training error 
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4. CONCLUSION 

This paper underlines measurable parameters that can be perceived using sensor networks in terms 
of intelligent information messages. Such parameters include eye blink pattern, emotions and heart rate, of 
which blink pattern is given the utmost importance. Resultant messages are then percolated and conditionally 
tagged to excerpt intelligent information, thus generating exigency alert. Also it highlights both the sufficient 
and necessary conditions required for alarm to be raised. Paper proposes an intelligent framework along with 
Perspicacious Information Retrieval Technique and the prognostic algorithm for detecting the exigency. The 
proposed Facial Emotion Recognition System recognizes 7 emotions labeled Neutral, Angry, Disgust, Fear, 
Happy, Sad and Surprise. The algorithm is validated through MATLAB and results depict the categorization 
of faces as neutral, surprise and happy depending on the expression. 
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